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Certain IBA operations are valid only over certain classes of service. AQP 1 

rejects a WQE for an operation that is not valid for the configured class of 2 

service. 3 

4 

Connection oriented service requires that the consumer initiate a commu- 
nication establishment procedure (connection setup) with the target node 
to associate the QPs and establish QP context prior to any QP operation, 6 
Actually, all service classes except for raw datagram need some form of 7 
communication setup to associate queue pairs. For reliable datagram ser- 8 
vice, the node performs a communication establishment process to asso- g 
date an end-to end (EE) context (explained later) with each iarget node. ^ 
All QPs configured for Reliable Datagram service use established EE con- 1 
texts and the work request specifies which EE context to use for that op- 
eration. 12 

13 

Raw Datagrams are similar to unreliable datagrams, except that the 14 

source QP does not know the identity of the QP that will receive and pro- 1 5 

cess the message. Raw datagrams allow for routers that forward raw da- 16 
tagram packets to non IBA destinations on a disparate fabric (such as a 
LAN or WAN) that has no equivalent of a QP. There are two types of raw 
datagrams, IPv6 and Ethertype. IPv6 raw datagrams contain a global 

routing header and the packet pay load contains a transport protocol ser- 1 $ 

vice data unit as identified in the global routing header. An Ethertype raw 20 

datagram contains an Ethernet Type field and the packet payload contains 21 

a transport protocol service data unit as identified in the Ethernet Type 22 

field. 23 

24 

IBA defines both channel (send/receive) and memory (RDM A) semantics. 

Raw datagram and Unreliable Datagram services do not support memory 2 ^ 

semantics. 26 

27 

3.5.3 Keys 28 

IBA uses various keys to provide isolation and protection. Keys are values 29 

assigned by an administrative entity that are used in messages in various 30 

ways. The keys themselves do not provide security since the keys are 3^ 

available in messages that cross the fabric and thus any entity that can ^ 

get to the interior of the fabric can ascertain key values. IBA does place ^ 
restrictions on how applications can access certain keys. 

The keys are: 35 

36 

• Management Key (M_Key): Enforces the control of a master subnet 37 
manager. Administered by the subnet manager and used in certain 33 
subnet management packets. Each channel adapter port has a ^ 
M_Key that the SM sets and then enables. The SM may assign a dif- 
ferent key to each port. Once enabled, the port rejects certain man- 
agement packets that do not contain the programmed M_Key. Thus 41 
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only a SM with the programed M_Key can alter a node's fabric con- 1 

figuration. The SM can prevent the ports MJ<ey from being read as 2 

long as the SM is active. The port maintains a time-out such that the 3 

port reverts to an unmanaged state if the SM fails. There is one 4 
M_Key for a switch. 

* Baseboard Management Key <BJ<ey): Enforces the control of a 6 
subnet baseboard manager. Administered by the subnet baseboard 
manager and used in certain MADs. Each channel adapter port has a 

B_Key that the baseboard manager sets. The baseboard manager 0 

may assign a different key to each port. Once enabled, the port re- 9 

jects certain management packets that do not contain the pro- 1 0 

grammed B_Key„ Thus only a baseboard manager with the -n 

programed B_Key can alter a node's baseboard configuration. The ^ 

baseboard manager can prevent the port's B_Key from being read as ^ 
long as the baseboard manager is active. The port maintains a time- 
out such that the port reverts to an unmanaged state if the baseboard 
manager fails. There is one B_Key for a switch. 

Partition Key (P„Key): Enforces membership. Administered through 

the subnet manager by the partition manager (PM), Each channel 17 

adapter port contains a table of partition keys which is setup by the 1 & 

PM. QPs are required to be configured for the same partition to com- 19 

munlcate (except QPO, QP1> and ports configured for raw data- 20 

grams) and thus the P_Key is carried In every IB transport packet. 21 

Part of the communication establishment process determines which 22 
P_Key that a particular QP or EEC uses. An EEC contains the P_Key 
for Reliable Datagram service and a QP context contains the P_Key ^ 

for the other IBA transport types. The P_Key in the QP or EEC is 24 

placed in each packet sent, and compared with the P_Key in each 25 

packet received. Received packets whose P_Key comparison fails 28 

are rejected. Each switch has one P„_Key table for management mes- 27 

sages and may optionally support partition enforcement tables that 2 $ 

filter packets based on their P_Key. ^ 

Queue Key (Q_Key): Enforces access rights for reliable and unreli- 3 q 

able datagram service (RAW datagram service type not included). ^ 
Administered by the channel adapter. During communication estab- 
lishment for datagram service, nodes exchange QJ<eys for particular 

queue pairs and a node uses the value it was passed for a remote ^ 

QP in all packets It sends to that remote QP Likewise, the remote 34 

node uses the Q_Key it was provided. Receipt of a packet with a dif~ 35 

ferent Q_Key than the one the node provided to the remote queue 35 
pair means that packet is not valid and thus rejected. 

Q_Keys with the most significant bit set are considered controlled 3$ 

Q_Keys (such as the GSI Q_Key) and a HCA does not allow a con- 39 

sumer to arbitrarily specify a controlled Q„Key. An attempt to send a 4Q 

controlled Q Key results in using the Q Key in the QP context. Thus MA 
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the OS maintains control since it can configure the QP context for the 1 

controlled Q_Key for privileged consumers. 2 

Memory Keys (LJ<ey and R_Key); Enables the use of virtual ad- 3 

dresses and provides the consumer with a mechanism to control ao 4 

cess to its memory. These keys are administered by the channel 5 

adapter through a registration process. The consumer registers a re- g 
gion of memory with the channel adapter and receives an L_Key and 
R_Key. The consumer uses the L_Key in work requests to describe 

local memory to the QP and passes the R_Key to a remote consumer 8 

for use in RDMA operations. When a consumer queues up a RDMA © 

operation it specifies the R_Key passed to it from the remote con- 1 0 

sumer and the R_Key is included in the RDMA request packet to the n 

original channel adapter. The R_Key validates the sender's right to ^2 
access the destination's memory and provides the destination chan- 
nel adapter with the means to translate the virtual address to a physi- 
cal address. 



3.5.4 Virtual Memory Addresses 



13 
14 
15 
16 

IBA is optimized for virtual addressing. That is, an IBA consumer uses vir- 17 
tual addresses in work requests and the channel adapter is able to con- ^ Q 
vert the virtual address to physical address as necessary. For this to 
happen, each consumer registers regions of virtual memory with the " 
channel adapter and the channel adapter returns 2 memory handles 20 
called L__Key and R__Key to the consumer. The consumer then uses the 21 
L__key in each work request that requires a memory access to that region. 22 
See 3.5.3 for description of L_Key usage. 23 
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Memory Registration provides mechanisms that allow IBA consumers to 
de-scribe a set of virtually contiguous memory locations or a set of phys- 
ically contiguous memory locations to allow the HCA to access the 26 
memory as a virtually contiguous buffer using virtual addresses. 27 

28 

IBA also supports remote memory access (RDMA) that permits a remote 29 

consumer to access that registered memory. For RDMA, the consumer 3$ 
passes the R_KEY and a virtual address of a buffer in that memory region 
to another consumer. That remote consumer supplies that R_Key in its 
RDMA WQEs that will access memory in the original node. See 3.5.3 for 

detailed description of R_Key usage. 3 ^ 

34 

3.5.5 Protection Domains 35 

Not only does memory registration allow the use of virtual memory ad- 3 ® 

dressing, but it also provides an increased leve! of protection against in- 37 

advertent and unauthorized access. 38 

39 

Since a consumer might communicate with many different destinations 4Q 
but not wish to let all those destinations have the same access to its reg- 
istered memory, IBA provides protection domains. Protection domains ^ 
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node. After the operation is completed, the I/O unit then uses channel se~ 1 
mantles to push an I/O completion message back to the processor node. 2 



3.6.1 Communication Interface 



Specifically, the operations supported for Send Queues are: 



3 
4 

"Channel adapter* is the term that identifies the hardware that connects a 5 
node to the ISA fabric (and includes any supporting software). The Q 
channel adapter for a processor node is called a "host channel adapter 
(HCA) and a channel adapter in an I/O node is a " target channel adapted 
(TCA), A consumer communicates through one or more "queue pairs" b 
(QP). An HCA typically supports hundreds or thousands of QPs while a 9 
TCA might support less than ten QPs. 10 

11 

It is the QP that is the communication interface. The user initiates work re- 1 2 
quests (WR) that causes work items, called WQEs, to be placed onto the 
queues and the channel adapter executes the work item. 
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Send Buffer - a channel semantic operation to push a local buff- 17 
er to a remote QP's receive buffer. The Send WR includes a gath- ^ q 
er list to combine data from several virtually contiguous local 
buffer segments into a single message that is pushed to a remote 
QP's Receive Buffer. The local buffer's virtual addresses must be 
in the address space of the consumer that created the local QP. 21 

22 

RDMA Read — a memory semantic operation to read a virtually 
contiguous buffer on a remote node. The RDMA Read operation 3 
reads a virtually contiguous buffer on a remote endnode and 24 
writes the data to a local memory buffer 25 

Like the Send operation, the local buffer must be in the address 26 
space of the consumer that created the local QP. 

op 

The remote buffer must be in the address space of the remote con- 
sumer owning the remote QP targeted by the RDMA Read. 29 

30 

RDIvlA Write — a memory semantic operation to write a virtually 
contiguous buffer on a remote node. The WR contains a gather ^1 
list of local buffer segments and the virtual address of the remote 32 
buffer into which the data from the local buffer segments are writ- 33 
ten. 34 

Like the Send WR, the local buffer must be in the address space 35 
of the consumer that created the local QP. 36 

The remote buffer must be in the address space of the remote con- 3? 
sumer owning the remote QP targeted by the RDMA Write. 38 

Atomic — a memory semantic operation to do an atomic opera- ^9 
tion on a remote 64 bit word. The Atomic operation is a combined 40 
Read, Modify, and Write operation, 41 

42 
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